1. Field of the Invention
The present invention relates to an image pickup apparatus and recording medium suitable for a digital camera or the like having a multiple exposure function.
2. Description of the Related Art
As a camera shake reduction technology suitable for an electronic still camera, a multiple exposure technology for obtaining an image free from a camera shake by performing continuous imaging of a plurality of images at a high shutter speed which does not cause a camera shake, and synthesizing the plurality of images while making registration thereof in a step after imaging has been considered.
To superimpose a plurality of images, motion vectors between the plurality of images are detected by use of information on the plurality of images at a pixel level, and positional displacements between a plurality of frames are corrected based on the detected motion vectors.
As means for detecting the motion vectors of the images, a block matching method and a correlation method based on a correlation operation have been conventionally known.
The block matching method is a method of dividing an input image signal into a plurality of blocks in a right size (for example, 8 by 8 matrix of pixels), calculating differences between each block, i.e., pixels in a fixed range, and the corresponding block in a previous field or a previous frame, and searching for a block in the previous field or the previous frame, in which the sum of the absolute values of the differences is minimum. The relative displacement between images, i.e., positional correlation between images, is indicated by a motion vector of the block.
In block matching, a block of high correlation is searched for by evaluation based on a squared difference SSD and an absolute value of a difference SAD. In a reference block region I and a target block region I′ on which matching is performed, supposing that pixel positions are pεI and qεI′ (p and q each represents a two-dimensional quantity, I and I′ each represents a two-dimensional region, and pεI means that coordinate p is included in region I), and pixel levels are Lp and Lq, the squared difference SSD and the absolute value of the difference SAD are defined as
                              SSD          ⁡                      (                          I              ,                              I                ′                                      )                          =                              ∑                                          p                ∈                                                                  ⁢                I                            ,                              q                ∈                                  I                  ′                                                              ⁢                                    (                              Lp                -                Lq                            )                        2                                              (        1        )                                          SAD          ⁡                      (                          I              ,                              I                ′                                      )                          =                              ∑                                          p                ∈                I                            ,                              q                ∈                                  I                  ′                                                              ⁢                                                Lp              -              Lq                                                                      (        2        )            
When the squared difference SSD and the absolute value of the difference SAD are both small, correlation is evaluated as high.
In the correlation method based on a cross-correlation operation, Ave (Lp) and Ave (Lq), which are a mean value of pixels pεI included in matching reference block region I and a mean value of pixels qεI′ included in matching target block region I′, are calculated, and a difference between each mean value and a pixel value included in each block
                                          Lp            ′                    =                                                    LP                -                                  Ave                  ⁡                                      (                    Lp                    )                                                                                                                    1                    n                                    ⁢                                                            ∑                                              p                        ∈                        I                                                              ⁢                                                                  (                                                  Lp                          -                                                      Ave                            ⁡                                                          (                              Lp                              )                                                                                                      )                                            2                                                                                            ⁢                          |                              p                ∈                I                                                    ,                                  ⁢                              Lq            ′                    =                                                    Lq                -                                  Ave                  ⁡                                      (                    Lq                    )                                                                                                                    1                    n                                    ⁢                                                            ∑                                              p                        ∈                        I                                                              ⁢                                                                  (                                                  Lq                          -                                                      Ave                            ⁡                                                          (                              qL                              )                                                                                                      )                                            2                                                                                            ⁢                          |                              p                ∈                                  I                  ′                                                                                        (        3        )            is calculated to obtain
                    NCC        =                  ∑                                    Lp              ′                        ⁢                          Lq              ′                                                          (        4        )            thereby evaluating a block whose NCC is large as a block of high correlation, and determining the displacement between block I and block I′ of the highest correlation as a motion vector.
When a picked-up subject, i.e., a subject included in an image, is stationary, the motion in each region is consistent with the motion of the entire image, and the correlation operation is performed on a block in an arbitrary fixed position, and a motion vector of the block is calculated.
There are some cases in which a motion vector with high reliability cannot be obtained due to noise or a condition in which a block is on a flat part or on an edge part of a structure that is larger than the block. To preclude such cases, Jpn. Pat. Appln. KOKAI Publication No. 8-163573 (Patent Document 1) and Japanese Patent No. 3164121 (Patent Document 2) suggest a technology for performing reliability determination when calculating a motion vector.
When a picked-up subject, i.e., a subject included in an image, includes a plurality of motions, it becomes necessary to calculate a motion vector of the entire image for the purpose of blur correction, for example. To achieve this object, Jpn. Pat. Appln. KOKAI Publication No. 8-251474 (Patent Document 3) suggests a technology for dividing a subject into a plurality of regions, selecting a main region from the plurality of regions by use of a region selection means based, for example, on the magnitude of the motion vector and the magnitude of the region, and determining the motion of the main region as the motion of the entire image.
Patent Document 3 discloses the following configurations:                The region selection means selects a region covering the largest area from the plurality of regions.        The region selection means selects a region of the smallest motion vector from the plurality of regions.        The region selection means selects a region having the largest area overlapping the region previously selected from the plurality of regions.        The region selection means selects one of the region covering the largest area, the region of the smallest motion vector, and the region having the largest area overlapping the region previously selected.        
Image processing technology related to a main region and a background, which is a region other than the main region, is disclosed in Jpn. Pat. Appln. KOKAI Publication No. 2003-125289 (Patent Document 4) and Jpn. Pat. Appln. KOKAI Publication No. 2007-074031 (Patent Document 5).
Patent Document 4 discloses technology for performing softening processing on a background when synthesizing a main region and a background. Patent Document 5 discloses technology for performing panning filter processing on separated backgrounds and then performing synthesis processing so as to produce a panning effect.
When blur correction is performed placing importance on correction of the motion in the main region as described in Patent Document 3, there arises a problem that, in image processing for superimposing a plurality of images, displacements of images of a subject (image overlapping) are conspicuous in the background part which shows a motion different from the motion of the main region.
When blur correction is performed placing importance on the main region, images may be picked up under a condition that reduces the image overlapping. However, complicated setting by a user needs to be simplified.
Further, as shown in Patent Document 5, image processing on an image-overlapping portion may be performed by making region segmentation, calculating a motion vector of each region, performing registration placing importance on a main region, and performing filter processing on a region other than the main region.
However, to perform image processing to the extent that the image is not unnatural at the boundary between the main region and a region other than the main region, a region must be divided into blocks of a small pixel unit, therefore image processing needs to be improved. Consequently, the calculation becomes large.
In view of above-described situations, the present invention has been made for the purpose of providing an image pickup apparatus and recording medium capable of easily forming a multiple exposure image in which image overlapping is inconspicuous in an unfocused image region.